fix(gitlab): add missing repos scope in project_mapping#8743
Merged
Conversation
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742
dosubot
Bot
added
size:M
klesh
approved these changes
Mar 2, 2026
Contributor
klesh
left a comment
klesh
left a comment
There was a problem hiding this comment.
LGTM.
Thanks for your contribution.
This was referenced Mar 3, 2026
spiffaz
added a commit
to spiffaz/incubator-devlake
that referenced
this pull request
Mar 3, 2026
…pesV200 When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
spiffaz
added a commit
to spiffaz/incubator-devlake
that referenced
this pull request
Mar 3, 2026
…in makeScopeV200 When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
klesh
pushed a commit
that referenced
this pull request
Mar 5, 2026
…in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749
lrf-nitro
pushed a commit
to AkerBP/nitro-devlake
that referenced
this pull request
Mar 9, 2026
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
lrf-nitro
pushed a commit
to AkerBP/nitro-devlake
that referenced
this pull request
Mar 9, 2026
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
klesh
pushed a commit
that referenced
this pull request
Mar 12, 2026
* feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes #8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in #8743. Closes #8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in #8743. Closes #8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com>
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Mar 26, 2026
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Mar 26, 2026
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Mar 26, 2026
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Mar 26, 2026
* feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com>
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Apr 9, 2026
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Apr 9, 2026
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Apr 9, 2026
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
la-tamas
pushed a commit
to archfz/incubator-devlake
that referenced
this pull request
Apr 9, 2026
* feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com>
3 tasks
zvika-finally
pushed a commit
to zvika-finally/incubator-devlake
that referenced
this pull request
Apr 30, 2026
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
zvika-finally
pushed a commit
to zvika-finally/incubator-devlake
that referenced
this pull request
Apr 30, 2026
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
zvika-finally
pushed a commit
to zvika-finally/incubator-devlake
that referenced
this pull request
Apr 30, 2026
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
makeScopeV200did not create areposscope whenscopeConfig.Entitieswas empty or only containedCROSS, causingproject_mappingto have notable='repos'rowproject_mappingwithtable='repos'makeScopesV200(lines 155-160) by:plugin.DOMAIN_TYPESDOMAIN_TYPE_CROSSto the repo scope conditionCloses #8742
Changes
backend/plugins/gitlab/api/blueprint_v200.go- two-line fix inmakeScopeV200backend/plugins/gitlab/api/blueprint_V200_test.go- two new test cases:TestMakeScopesWithEmptyEntities- verifies empty entities default to all domain typesTestMakeScopesWithCrossEntity- verifies CROSS entity triggers repo scope creationTest plan
TestMakeScopes,TestMakeDataSourcePipelinePlanV200)mericodev/lake-builder:latestDocker container (same as CI)Note
The same issue also affects Bitbucket (missing empty default) and Azure DevOps (missing both). If this fix is accepted, I am happy to submit follow-up PRs for those plugins.